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ABSTRACT 

This investigation examined the influence of sample 
size on different goodness-of-f it indices used in confirmatory factor 
analysis (CFA). The first two data sets were derived from large 
normative samples or responses to a multidimensional self-concept 
instrument and to a multidimensional instrument used to assess 
students' evaluations of teaching effectiveness. In the third set, 
data were simulated and generated according to the model to be 
tested. In the fourth, data were simulated and generated according to 
a three-factor model that did not have a simple structure. Twelve fit 
indicators were used to assess goodness-of-f it in all CFAs. All 
analyses were conducted with the LISREL V package. One-way ANOVAs and 
a visual inspe,ction of graphs were used to assess the sample size 
effect on each index for the four data sets. Despite the 
inconsistency of the findings with previous claims, the results are 
consistent with the observation that the amount of random, 
unexplained variance varies inversely with sample jsize. Appendices 
include a set of computed statements, an explanation and listing of 
the 12 goodness-of-fit indicators, a bibliography, a table of 
results, and figures showing sample size effect. (Author/LMO) 
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Goodness-of-f it Indices In Confirmatory Factor Analysis: 
The Effect of Sample Size 

ABSTRACT 

The present investigation examines the influence of sample size on 
different goodness-of-f it indices used in confirmatory factor analysis (CPA). 
Contrary to Bentler and Bonett (1980) i their incremental fit index was 
substantially affected by sample size. Contrary to Joreskog and Sorbom (1981), 
their goodness of fit indices provided by LISREL were substantially affected 
by sample size. Contrary to Hoelter (1983), his critical N index was 
substantially affected by sample size. Of the 12 indices considered, only the 
Tucker-Lewis index and a conceptually similar new index were relatively 
independent of sample size, and these results were consistent across two real 
and two simulated sets of data. Despite the inconsistency of these findings 
with previous claims, the results are consistent with the observation that the 
amoufit of random, unexplained variance varies inversely with sample size. 
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6oodness-of-f it Indices In Confirmatory Factor Analysis: 
The Effect of Sample Size 

The purpose of the present investigation is to examine the influence of 
sample size on different goodness-of-f it indicators used in confirmatory 
factor analysis (CFA). While the present investigation is limited to CFA, the 
problems, issues and results also generalize to the analysis of covariance 
structures. The advantages of the use of CFA are mil known and numerous 
introductions to the LISRt^L approach used in the present investigation are 
available elsewhere (e.g., Sagozzi, 1980; Joreskog It Sorbom, 1981; Long, 1983; 
Harsh tc Hocevar, 1985j Pedhauzur, 1982). Briefly, in CFA, responses to p 
observed variables by N subjects are summarized by a (p x p) sample covariance 
matrix and it is hypothesized that the corresponding population covariance 
matrix can be summarized by q true but unknown parameters (Bentler & Bonett, 
1780). The q parameters in the present investigation are the factor loadings, 
the factor variances and covariances, and the error/uniquenesses. To the 
extent that the inferred population covariance E derived from these parameters 
corresponds to the observed sample covariance matrix S, the model is 
supported. The problem of goodness of fit is how to decide whether E is 
sufficiently similar to S to justify the conclusion that a specific model 
adequately fits a particular set of data (Hoelter, 1983). 

In the maximum likelihood approach to CFA a discrepancy or loss function 

is iclnimized with respect to the q parameters such that its value approaches 

zero as the S and E become identical (Bentler tc Bonett, 1980; Joreskog & 

Sorbom, 1981). If all the observed variables have a multivariate normal 

distribution, if the sample size is iarne, and if the model is correct, then N 

2 

- 1 times the minimum value of the loss function can be interpreted as a X 
test statistic with degrees of freedom (df) e|ual to .5 x p x (p H - q. As 

typically used the model is rejected if the X is large relative to the df, 

2 

and accepted if the X is nonsignificant or small. However, Bentler and Bonett 
(1980) warn that the probability of detecting a false model increases with N 
even when the model is minimally false (i.e., differences between E and S are 
trivial) so that for very large sample sizes nearly all models are rejected. 
(It is important to note that for a true model the expected value of X is 
equal to the df and does not vary with sample size.^ Because of the 
apparently restrictive assumptions underlying the X test statistic and 
because of the power of the test for large sample sizes, Marsh and Hocevar 
(1985) concluded that **most applications of confirmatory factor analysis 
require a subjective evaluation of whether or not a statistically significant 
chi-square is small enough to constitute an adequate fit" (p. 567), that this 
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subjectivity undermines soflie of the rigor that is possible with CFA, and that 
"this issue will continue to be an important one in the future development of 
this statistical procedure" (p. 568). 

Bentler and Bonett (1980) note the dubious logic of inferring support for 
a model from a nonsignificant X (i.e., attempting to prove the null 
hypothesis) since the X can be made small simply by reducing the sample size. 
Because of this influence of sample^size, a poor fit based on a small sample 
size may result in nonsignificant X , whereas a good fit based on a large 
sample size may result in statistically significant X . The substantial 
influence of sample size on X for a false model may lead tc a counter- 
productive practice in CFA. In order to obtain a nonsignificant X , or at 
least c?.?» that is acceptably low, researchers may be tempted to limit analyses 
to a small number of cases, or if their sample size is large to analyze only a 
subsample of their data. For example, Bi/ntler and Bonett (1980, p. 5?J) assert 
that "one's favorite laodel will stand the best chanca of being accepted when 
tested against the data of small samples" and Fornell (1983, p. 446) suggests 
that one could "make sure the sample size is not large enough to reveal a 
difference between the model and the data". As emphasized by these autha-s, 
and by common sense, such a practice is counter-productive and Hoelter (1983, 
p. 328) stated that: 

"sacrificing the power of a test by utilizing small sample sizes simply 

blinds the researcher to significant differences between a model and the 

data' and 

"testing models with large samples is ali^ays desirable, and the question 
that needs to be addressed deals with how well a model aBBCSJiiiates the 
observed data rather than whether of not the model fits the data." 
Because of the substantial influence of sample size on X , researchers have 
developed a variety of goodncss-of-f it indices that they claim to be 
unaffected by sample size, and these are among the fit indices considered in 
the present investigation. 
Goodness gf Fit Indicators^ 

and X^^df Ratigj. These two indices continue to be the most frequently 
used goodness-of-fit indicators. As noted above, the X for a false model 
varies directly with sample size, but the X for a true model does not. In 
CFA the df does not vary with the sample size, so that the effect of sample 
size on the X /df must necessarily be the same as for the X . Hence, while 
these indicators do not vary with sample size for a true model, they are 
substantially influenced by sample siza when the model is false and this 
dependence on sample size is larger for poorer models. 
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lD£Cg!5fQ48l Eii 10dt£§ii. Bentl ar and BooKtt (1980) proposed that valuable 
information could be obtained by comparing the ability of nested models to fit 
the same data. In particular, for CFA it is useful to compare the fit ot a 
null model in which all the p variatles are assumed to be uncorrelated with 
the fit of the proposed model. If the fit of a null model is reasonable, 
because the sample size is small or because the measured variables are 
relatively uncorrelated, then the fit rf target model will automatically be 
reasonable. However, if the increment in the goodness-of-fit is small, then 
there may be no basis of support for the model. Bentler and Donett (1980) 
presented two incremental fit indices. First they described the Tucker-Lewis 
Index (TLX) and a more general version of this index. Second, they proposed an 
alternative index, called the Bentler-Bonett Index (BBI) for purposes of the 
present investigation. They specifically noted that these indices are useful 
for comparing the fit of a particular model across samples that have unequal 
sires. They cautioned that the absolute value of these indices may be 
difficult to interpret, but that v«iups of less than .9 usually mean that the- 
model can be improved substantially. Much of the value of these indices is 
based on the assumption that they are indepmdent of sample size, and this 
assumption will be tested in the present investigation. 

Other at iDdicgs cgmeytg^ LimU Joreskog and Sorbom (1981) describe 
three other indices that are computed by LISRELt the goodness-of-fit index 
(GFI), the adjusied 6FI (A6FI), and the root mean square residual (RMS). They 
describe GFI as "a measure of the relative amount of variances and covariances 
jointly accounted for by the model" and assert that "unlike GFI is 
independent of the sample size" while the adjusted GFI (AGFI) "corresponds to 
using mean squares instead of total sums of squares" (Joreskog & Sorbom, 1981, 
p. I. 40-41). The RMS, based on the differences between the elements in the 
observed and inferred covariance matrices, is easily interpreted when 
correlation matrices are analyzed. For the examination of covariance matrices 
RMS still has a lower-bound of zero but its upper bound is indeterminant. Thus 
RMS must be interpreted in relation to the size of the variances and 
covariances of the measured variables, and cannot be compared across 
applications based on different variables. Joreskog and Sorbom suggest that 
GFI and AGFI will generally fall between 0 and 1, but that it is possible for 
them to be negative. Again, much of the value of these indicators is based on 
the assumption that they are independent of sample size, and this assumption 
will be tested in the present investigation. 

ecitical Hoelter (1983, p. 528) argued that "rather than ignoring or 
completely neutralizing sample size we can estimate the size that a sample 



must rec^ch in order to accept the fit of a given nKidel on a statistical basis. 
This estimate! referred to he^^e as 'critical N' (CN), allows one to assess the 
fit of a eiodel relative to identical hypothetical models estimated with 
different sample sizes. Hoelter cautioned that no firm basis could be offered 
as to what constituted an adequate fit, but he suggested that a value of 200 
was a reasonable starting point for suggesting that differences between the 
model and data may be unimportant. The usefulness of CN also rests on the 
assumption that its value is independent of sample sizei and this assumption 
will be testCfd in the present investigation. 

Itl§ §9kSl ^DA iQbCOfi&Sdi SBe!:Si£b& purposes of the present 
investigation only two model S| a null model and a target model i are fit to 

each covariance matrix. However, as emphasized by.Sobel and Bohrnstedt 
(1965), by Bentler and Bonett (1980), and by Harsh and Hocevar (1984; 1985) in 
most substantive applications a variety of nontrivial models can be generated 
on the basis of previous research or the logic of the data. Often these 
ftontrivial models will differ from each other according to strict statistical 
criteria, but differences may not be of practical importance. 6oodness-of-f it 
indicators in which each of the competing nontrivial models is compared to the 
null model may, perhaps, provide an externally meaningful, well defined, 
absolute scale for determining whether -he statistically significant 
differences are of practical importance (e.g^. Marsh & Hocevar, 1984; 1985). 

Sobel and Bonrnstedt (1985) contend that the use of the null model io 
determine one end-*point for incremental fit indices is only appropriate for 
exploratory studies in which nontrivial alternative models are unavailable. 
Instead, they argue that a more parsimonious nontrivial model shoulJ be used 
to define the lower-end of the goodness-of-f it index. However, there are two 
important problems with their approach. First, the use of the null model to 
define one end-»point of goodness-of-f it indices does not preclude the 
comparison of alternative nontrivial models and need not be limited to 
exploratory studies (e.g.. Marsh, 1985; Marsh tc Hocevar, 1984; 1985). Second, 
and more importantly, if their recommendation were followed then the scale of 
their goodness-*of-*f it indices would be arbitrarily determined by the 
idiosyncratic choice of nontrivial models in a particular study, thus 
precluding inferences of practical significance of differences between 
competing nontrivial models on a nonarbitrary scale. Furthermore, for the 
incremental fit indices they considered and considered here, the value 
obtained from the Sobel /^ohrnstisdt approach is just a linear transformation of 
the one based on the <iull model. Whether the promise of an externally 
meaningful, well-defined, absolute scale for goodness-of-f it indicators can be 
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fulfilled by existing fit indices is an empirical question that is addressed 

in part by the present study, but the promise cannot be fulfilled if the 

Sobel/Bohrnsterit approach is adopted. It should also be poted that if the 

sample size affects goodness of fit as used in the present approach based on 

the null model, it m^ist also affect goodness of fit as defined with the 

Sobel /Bohrnsted t approach . 

Ids PC§S§Qt iQveitiSSUSQiL 

For purposes of the present investigation, it is proposed that an ideal 

goodness-of-f it index should! 

1) be relatively independent of sample size; 

2) be valid, that is accurately and consistently reflect differences in 
goodness of fit for competing models of the same data (e.g., nested models) 
and for the same model applied to different data; 

3) vary along an externally meaningful, well-defined, absolute continuum 
so that its value can be easily interpreted. 

The present investigation emphasizes the examination of the first 
criterion, but its violation would also undermine the second and particularly 
the third criteria. In order to examine the first criterion the same three- 
factor, simple-structure model was fit to data from four data sets. The first 
two data sets, based on real data, were derived from large normative samples 
of responses to a multidimensional self-concept (SC) instrument and to a 
multidimensional instrument used to assess students' evaluations of teaching 
effectiveness iSET). Based on the a priori design of each instrument, the 
selection of variables, and the results of jjrevious research, a three-factor 
model is reasonable. However, only the SC data approximates a simple structure 
so that it should be better fit by the model than should the SET data. 

The third and fourth data sets are based on simulated data. In the 
third, data were generated according to the model to be tested, so that E and 
S differ only due to random chance. In the fourth, data were generated 
acccx-ding a three-factor model that did not have a simple structure. Hence, 
goodness of fit indices should be better for the third data set than for the 
fourth, and also better for the third than either of the first two. 

For each of the four data sets, ten random samples were generated with 
sample sizes of 25, 50, 100, 200, 400, 800, and 1600, and the same three- 
factor model was fit to 280 covariance matrices derived from the four data 
sets. To the extent that the values of a particular index of fit are similar 
across the seven sample sizes, then there is support for its independence of 
sample size. 

Method 
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Ib§ Modgl and Ana^ysM^ 

All analyses were conducted with the commercially available LISREL V 
package (Joreskog & Sorbom, 1981) • The target model in each of the analyses 
contained 21 estimated parameters: six factor loadings (in LAMBDA Y), three 
factor variances (in PSD, three factor covariances (in PSD, and nine 
error/uniquenesses (in THETA EPSILON). The first measured variable for each 
factor was designated to be a reference indicator and given a factor loading 
of 1.0, while loadings for the other two variables were estimated. Hence the 
df (.5 X 9 X 10 - 21 « 24) was constant for all the analyses. In addition to 
the target model, a null model was tested for each covariance matrix such that 
the reproduced covariance matrix was a diagonal matrix of variances and the 

nine measured variables were posited to be uncorrelated. The df for the null 
model (.5 X 9 X 10 - 9 « 36) was also constant for all the analyses. The same 
null and hypothesized models were tested for each of 280 covariance matrices 
described below. (Also, in order to test the generality of findings to be 
described latter, one additional set of analyses was conducted on a 14- 
variable, 4-factor model described latter). 
Jhe Data.. 

Tne SamQle Sizes*. The seven sample sizes to be ?:onsidered in the present 
investigation, 25, SO, 100, 200, 400, 800 and 1600, were selected to span the 
rwge of sample sizes typically considered in CPA. Hoelter (1983) and Bentler 
and Bonett (1980) each considered a sample size of 23 for a similar purpose, 
and so 25 appeared to be a reasonable lower limit. The upper limit of 1600 
was arbitrary, but it is apparently larger than the sample sizes typically 
used in CPA. (Also, in order to test the generality of findings to be 
described latter, one additional set of analyses was conducted nith a sample 
of 32,000). 

igI£z£QO£SB^ liEl ^ata^ The SC data came from the normative archive of 
over 4,000 sets of responses to the Self Description Questionnaire (Marsh, 
1986). Por purposes of the present investigation the first three indicators 
were selected from the Physical Ability, Reading, and Math scales. Previous 
research (e.g.. Marsh & Hocevar, 1985; Marsh, Smith te Barnes, 1985) suggests 
that the simple structure model should provide a reasonable fit for this data. 
A random sample of 3175 cases was divided into sets of 25, SO, 100, 200, 400, 
800, and 1600, and this process was repeated 10 times so that a total of 70 
covariance matrices were created. The hypothesized and null models were fit to 
the a 9 X 9 covariance matrix derived from this d&ta set. 

Studentsl e^alyatigQg g£ t§3£biQa I§ill£. This data came from the cet of 
4,471 sets of class-average responses to the Students' Evaluation of Educational 
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Quality instrument that was factor analyzed by Marsh (1983} 1984)* For purposes 
of the present investigation the first three items were selected from the 
Leaning/ Value, Organization, and Kc?rkload/Dif f iculty scales. Previous factor 
analyses suggest that while the three factors are well defined, several of the 
items load on more than one factor so that solution is not -sim;ile.- As with the 
SC data, the null and hypothesized model were fit to 70 covariance matrices. 

SiCBlfi ftcygtyre ftiSilitEtf ti§t§ igSSBl^ The nine measured variables were 
defined with the random number generator from the commercially available SPSS 
package (Hull & Nie, 1981). Each variable was defined to reflect only one 
factor and a normally distributed random error component, and the three 
factors were defined to be correlated (see Appendix I). A total a 31,750 cases 
were generated and divided into 70 sets of data such that each sample size was 
represented by 10 covariance matrices, and the null end hypothesized models 
we*e fit to the 70 covariance matrices. 

Gg<DBl§i gtryctyrg §ifiyl§tgd B§ta iQS§5ia. The nine measured variables were 
defined as with the SSSD except that six of the nine measured variables ~ two 
for each factor — were defined such that each should have had a small loading 
on one factor in addition to the one it was designated to reflect (see Appendix 
I). Again a total of 31,750 cases were generated and divided into 70 sets of 
data such that each sample size was represented by 10 covariance matrices, and 
the null and hypothe^sized models were fit to the 70 covariance matrices. 
&ggdn§§g si iQdlcatgrs^ ^ ^ 

Six goodness-of-fit indicators ~ X , X /df , GFI, AGFI, RHS, and CN — 

were examined that did not require results from a corresponding null model. 

Two forms of an incremental fit index described by Bentler and Bonett (1980) 

were used to reflect a scaled difference between the goodness of fits for the 

null and hypothesized models^ One form is illustrated by the BBI| the 

2 2 
difference in X for the Null and tested model divided by the X for the null 

model. Using this approach, new incremental fit indices (see Appendix ID 

2 

were derived from X /df (called MBl), RMS (called incremental RHS), and CN 

(called incremental CN). The range of values for these indices is strictly 

bounded by 0 and J. A second form is illustrated by the TLX} the difference in 
2 

X /df for the Mull and posited models is divided by the difference between the 
2 

X /df for the Null model and some ideal expected value — in this case 1.0. 
Using this approach a new incremental index was derived from the X (called 
MB2). For these indices, the range of expected values is 0 to 1.0. ^Technically 
it is possible for these indices to be slightly negative (if the X^/df for the 
null model is less than 1.0) or slightly larger than 1.0 (if the X /df for the 
target model is less than 1.0) but such occurrences will be very rare for real 
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data tested nith a priori ^.i.. The set of 12 fit indicator., a. defined in 
Appendix II, Ha. u.ed to ...... the goodne.. of fit in all 280 CPAs. 

Re.ults tt Discussion 
One-Hay ANOVAs (Table l> and a visual inspection of graphs (Figure 1, 
Here used to assess the .aepl. sire effect on each index for the four data 
.ets The effect of sa.pl. sire varies dra.atically depending on the index. Of 
the 12 indicM con.idered, only the TLI and the conceptually si.ilar MB2 are 
r-latively independent of saepie size. For the other 10 indices, th. ...pi. 

.iz. effect varie. nith^both the ...pie .ize and the data ..t. For 7 of these 
10 indice. - all but X , X /df, and CN - ««t of the ...pi. .i,. 
occur, for the .««li„t ...pie .ir«» (25, 50, 100 tc 200). Thf., pa.pl. .iz. 
•**nt for ill but TLI and HB2 ulno app.ar. to vary nith th. data b.ing 
considered, m particular, the sa.ple sir. .ffect for all the indic«. exc.pt 

. X /df, TLI and MB2 ar. substantially larger for th. SSSD data that was 
SLulated to^be^Ht^by_tf^el than for the SC and/or SET data. 
Insert Table 1 and Figure I About Here 
Though not the primary focus of thr^P^^^^e^tigation, the four data 
sets were constructed so «s to vary in their ability to be fit by the target 
•odel. In particular, the SSSD data were simulated to be best fit by the 
-del and the SC data were selected so as to be better fit bv the .odel than 
Here the SET data. This ordering is accurately refl.ct«i by X^ X^/df , GFI 
A6FI, CN, TLI and MB2, but not by BBI, MBl, incr««ntal RHS, incr««ntal CN ~ 
-11 of Which are incremental indices. (Note that the RMS was not considered 
"nee it is dependent on the size of variances and covariances in th. 
particular application and so is clearly not comparable across applications 
unless analyses are based on the correlation matrices.) It was also as«^ 
that the fit for the target model should always be better than for the 
corresponding null model. However, AGFI and occasionally SFI had negative 
values for the target model whereas the corresponding null model, always had 
positive values. Thus, at least for these cases, GFI and AGFI did not accurately 
reflect differences in the ability of the two models to fit the same data. These 
findings suggest that the five incremental indices other than TLI and MB2, and 
perhaps GFI and AGFI, in addition to being substantially influenced by sample 
«ze, may not validly reflect real differences in goodness of fit. 

Contrary to claims by Bentler and Bonett (1981), their BBI is 
substantially related to sample size in all four data sets. Contrary to claims 
by Joreskog and Sorbom (1981), their GTI and AGFI are both substantially 
affected by sample size in all four data sets (as well as the RHS that is also 
provided by LISREL). Contrary to claims by Hoelter (1983), his CN is 
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substantially related to sample size in all four data sets. Furthermore, All 
the other indices except for TLI and f1B2 were also substantially affected by 
sample sizes. While the empirical findings are clear, two important questions 
remain: Why are BBI, GFI, A6FI, RMS, and CN so substantially affected by 
sample size? Why are the TLI and MB2 relatively unaffficted by sample size? 

The substantial effect of sample size on BBI, GFI, A6FI, RMS, and CN is 
consistent with the otservation that the standard errors of the observed 
variances and covariances becomes smaller as sample sizes become larger. Data 
to be fit contains variance that can be explained by the target model 
(explained variance), systematic variance that cvinot be explained by the 
model (uniqueness), and random variance that cannot be fit by the model 
(error). The proportion of random error decreases systematically with sample 
size (i.e., standard errors become smaller), so that the proportion of 
variance that can be explained and the uniqueness mcst increase with sample 
size. The relation between explained and random variance is most clear for the 
SSSD data since it was created to have no uniqt:eness« Thus this explanation of 
the sample size effect can be examined most easily for this data. 

For the SSSD data, the RMS reflects only random variance and the amount 

of random variance is inversely related to sample size. Thus RMS also varies 

2 

inversely with sample size. The BBI is based on the ratio of the X s for the 

2 

null and target model, but for the SSSD data the X for the null model varies 

2 

substantially with sample size while the X for the target model does not. 

Thus BBI must also vary with sample size. CN is based on the unstated 

assumption that the proportion of random variance will not vary with sample 

size, but this assumption is false. For the SSSD data with n«2S CN is small 

(41) because the proportion of random variance is relatively large, whereas 

for nsi600 CN is much larger (3114) because the proportion cf random variance 

is relatively small. If the SSSD were tested with a sample size of 41 and the 

proportion of random variance was the same as it was in a sample size of 25, 

then Holter's claim of the independence of CN and sample size would be 

satisfied. However, as empirically demonstrated in the present investigation, 

the claim is false, and this is because the proportion of random variance will 

not be the same in samples of 25 and 41. CNote also that if a covariance 

matrix based on 1600 cases is used but LISREL is told that it is based on ciily 

2 ^ 
100 cases, a much smaller X will be obtained than if a covariance based on a 

random sample of 100 of the 1600 cases is used.] GFI and A6FI, as measures of 

the relative variance that is accounted by the target model, will also vary 

with sample size since the amount of random variance that is unexplained by 

the model varies with sample size. Hence, RMS, GFI, AGFI, CN, and BBI all vary 
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with sample size because the proportion of random variance varies with sample 
size. For real data the situation is complicated by the existence of 
uniqueness in addition to random and explained variance. Howeveri even though 
it may be difficult to separate random error and uniqueness, the proportion of 
random variance still depends on sample size so that the logic of the present 
explanation generalizes to these data as well. 

A comparison of the BBI and NBl with TLX and MB2 suggests why the first 
two are so much more affected by sample size. Both sets of indices have the 
same numerators (see Appendix II), but the denominator of BBI and MBl is based 

on the fit of the null model whereas the denominator of the TLI and nB2 is the 

2 2 

difference between the null model and the expected value of the X or X /df. 
In the present study these expected values — df for the X or 1.0 for the 
X /df — are constant, but the size of this constant in relation to the value 
for the null model varies with sample size. For small sample sizes th.^s 
constant is relatively larger compared to the value for the null model, and 
this provides some control for the sample size effect. Additional research is 
clearly warranted to further examine this suggested explanation and to test 
the generality of the findings in different applications. 

ItlS ginecalit^ gf ttje SamQli gjize g£f g£t^ 
Bas^d on findings of this study two features of the sample size effect 
are clearly illustrated. First, the sample size effect is small or negligible 

for TLI and MB2, but is substantial for the other 10 fit indices. Second, for 

2 2 

7 of these 10 indices — all expect X , X /df and CN — the sample size effect 
is weaker, though still statistically significant, for the larger sample sizes 
(400, 800 & 1600). However, the findings are limited to seven sample sizes 
and to a single target model, and these limitations to the generality of the 
findings are worrisome. Consequently, two additional sets of analyses were 
conducted to test the generality of these findings. First, one additional 
sample based on 32,000 cases from the CSSD data — 20 times the largest number 
of cases considered in the present investigation — was fit with the same 
target model . Second, a new data set like the CSSD was constructed for 14 
measured variables designed to reflect four factors, and the sample size 

2 

effect on the different fit indices was determined for this new data set . 

The purpose of the 32,000-case analysis was to determine the behavior of 
the fit indices when the sample size was extremely large. Although the 
interpretation of the values is difficult for many of the indices that are 

approaching their optimum values for nBl600, only the TLI and MB2 appeared be 

2 2 

unaffected by this large jump in sample size. The X and X /df for the 32,000 
cases were substantially larger (poorer) compared to values for 800 and 1600 

ERIC 13^ .V 
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casesi CN Has substantially larger (improved), and the other indices — 
except TLX and MB2 — were marginally improved (see footnote 1). These 
findings support the generality of the findings summarized earlier, but do not 
indicate whether or not the xndices will continue to change as sample size 
approaches infinity. 

The purpose of the new datA set with 14 variables was to explore the 
generality of the sample size effect found with the 9 variable model. While 
the sample size effect was statistically significant for all fit indices, the 
size of the effect was substantially smaller for the TLX and HB2 than for any 
other fit indices (see footnote 2). Furthermore, the group based on samples 
sizes of 25 was the only group to differ significantly from any other group for 
TLX and NB2, whereas nearly all possible pair-wise comparisons between different 
sampler sizes were statistically significant for the other indices. These 
results also support that the generality of the findings summarized earlier. 

Summary 

The promise of an externally meaningful, well-defined, absolute scale 
does not appear to be fulfilled by most of the goodness-of-f it indices 
considered in the present investigation. First, when the variables to be fit 
and the model to be tested are held constant, values for 10 of the 12 fit 
indices are substantially affected by sample size. Furthermore, this sample 
size effect cannot be easily characterized f^nce it varies depending on the 
particular index, the data set that is bei j considered, and the range of 
sample sizes being considered. Second, when the model to be tested and the 
sample size are held constant, the fit indices may not be comparable across 
different data sets. Xn particular, the SSSD data was constructed so that it 
would be best fit by the target model, but of the incremental fit indices, 
only the TLX and MB2 accurately reflected this difference. Since only the TLX 
and MB2 performed satisfactorily in this investigation, no absolute criteria 
of what value constitutes^ an acceptable fit seems justified for any of the 
other indices (e.g., the .90 suggested by Bentler & Bonett for BBX, the 200 

suggested by Hoelter for CN, or the value of 2.0 sofi^times suggested for 

2 

X /df). On the basis of this investigation it is recommended that: a) 
additional research is conducted with the TLX and MB2 to further test their 

characteristics; and, b) pending the outcome of further research, at least one 

2 

of these two indices should be used, along with the X test of statistical 
significance and the examination of parameter estimates in relation to 
substantive issues, to assess goodness of fit. 
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J Footnotes 

— A single 9x9 covariance matrix was derived from a new sample of 

32,000 cases for the CSSD data (see Appendix I) and was fit with the same 3- 

factor model as the other 280 covariance matrices. The values for the 12 

2 

goodness-qf-fit indices defined in Appendix II are as follows: X (3155), 
X /df (;31.5), 6FI (•970), ABFI (.943), RMS (.074), CN (395.1), BBI (.927), 
MBl (.890), incremental RMS (.774), incremfjr/cal CN (.896), TLI (.891), MB2 
(.928). Compared to indices for samples of 800 and 1600 with the same CSSD 
data (see Table 1), the values for the TLI and MB2 did not differ, values 

for 6FI, AGFI, RMS, BBI, MBl, incremental RMS, and incremental CN were 

. t% 2 2 

marginally improved, values for X and X /df were substantially poorer, and 

the value for CN was substantially improved. 
2 

— Procedures similar to those used for the CSSD data were used to 

generate 14 random variables that reflected 4 correlated factors. The first 

nine variables were defined as with the CSSD data, and the fourth factor was 

defined by five additional variables that were constructed to reflect it. 

The data was fit with a simple structure model even though the simulated 

structure was complex. For just this analysis, 10 sets of 14 variables were 

generated with sample sizes of 25, 100, 400, and 160O. LISREL was used to 

derive the 12 goodness-^f-f it indices for each of the 40 covariance 

matrices, and one-way ANOVAs and visual inspections were used to assess the 

2 2 

sample size effect. F-ratios for the 12 fit indices were: X (238), X /df 
(238), GFI (404), AGFI (403), RMS (153), CN (268), BBI (304), MBl (304), 
incremental RMS (91), incremental CN (322), TLI (20), MB2 (16). A student- 
Newman-Kuels (Hull & Nie, 1981) test of pair-wise differences for the TLI 
and MB2 indicated that the mean value based on f\^25 was significantly 
different from all other sample sizes, but that no other pair-wise 
differences were statistically significant. For each of the other 10 
indices, nearly all of the possible pair-wise differences were statistically 
significant. These statistical analyses, and inspection of plots similar to 
Figure 1, indicate that TLI and MB2 are substantially less affected hy 
sample size than any of the other 10 fit indices. 
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Appendix I 

The following set of compute statements were used with SPSS (Hull & Nie, 
1981) in order to create 31,750 sets of nine variables (xl - x9) to 
represent the SSSD and CSSD data sets. Subsequent CPA analyses were based on 
70 covariance matrices — 10 for each of 7 sample sizes — derived from 
these nine variables^. 

















compute 


V s 


■ normal (1) 




compute 


il 


s 


.2 


t 


V + normal 


(1) 


compute 


i2 




.4 


t 


V + normal 


(1) 


compute 


iZ 




.6 


t 


V + normal 


(1) 


compute 


xl 




.6 


t 


fl + normal 


(1) 


compute 


x2 




.7 


t 


fl + normal 


(1) 


compute 


x3 




.8 


* 


fl + normal 


(1) 


compute 


x4 




.6 


t 


f2 + normal 


(1) 


compute 


x5 




.7 


t 


f2 + normal 


(1) 


compute 


x6 




.8 


t 


f2 + normal 


(1) 


compute 


x7 




.6 


t 


f3 + normal 


(1) 


compute 


x8 




.7 


t 


f3 + normal 


(1) 


compute 


x9 


s 


.8 


t 


f3 + normal 


(1) 


CSSD Data 


Set 












compute 


V = 


= norAtal ( 1 ) 




compute 


fl 




.2 


t 


V + normal 


(1) 


compute 


f2 




.4 


t 


V + normal 


(1) 


compute 


f3 




.6 


t 


V + normal 


(1) 


compute 


xl 




.2 


t 


f2 + .6 * fl + normal 


compute 


x2 




.2 


t 


f3 + .7 * fl + normal 


compute 


x3 




.8 


t 


fl + normal 


(i) 


compute 


x4 




.2 


t 


fl + .6 * f2 + normal 


compute 


x5 




.2 


t 


f3 + .7 * f2 + normal 


compute 


x6 




.8 


t 


f2 + normal 


(1) 


compute 


x7 


s 


.2 


t 


fl + .6 * f3 + normal 


compute 


x8 




.2 


t 


f2 + .7 * f3 + normal 


compute 


x9 




.8 


t 


f3 + normal 


(1) 



:er4,c 



16 




Goodness of fit 14 
Appendix II 

A total of 12 goodness-of-fit indicators were considered in the 
present investigation, and are described below. The first six indices 
require information from tests of ally the hypothesized model, and are 
called stand-alone indices for purposes of this investigation. The 
rest are based on difference in goodness-of-f it for the hypothesized 
model and its corresponding null model, and are referred to as 
incremental indices. Incremental index were derived according to the 
form (x-y)/y where x refers to the goodness-of-f it for either the 
hypothesized model or the null model, whichever is expected to be 
largest, and y refers to the goodness of fit for the other. This set 
of indices is called form 1 incremental indices for purposes of the 
present investigation. A second set of incremental indices was 
derived according to the form + or - (x-y)/(y-I) where x and y refer 
to the goodness of fits for the target and null models respectively, I 
is an ideal or optimum value for x. These are called form 2 
incremental indices for purposes of the present investigation. 
i^iQd Alone Indices^ 
2 

1) X (see Joreskog & Sorbom, 1981) 

2 

2) X /df^ where df^ = C .5 x p x (p+1) ] - q 

3) GFI = 1 - C (tC (e"^ X S - I)^/(tC E s"S^ 
(see Joreskog & Sorbom (1981, p. 1.40). 

4) AGFI « 1 - Cp X (p+l)/2df^3 x (1 - 6FI) 

(see Joreskog & Sorbom (1981, p. 1.40). 

2 1/2 

5) RMS « C 2 E E (s - o ) / p x (p+1)] where 

i C i C 

s and o are elements in S and E 
iC iC 

(see Joreskog & Sorbom (1981, p. 1.41). 

1/2 2 2 

6) CN « CCz + (2 X df - 1) 1 /12 X X /<H-lll +1 where 

cri C T T 

2^^^^ the critical value from a normal curve table for a given 
probability level — 1.96 in the present investigation (see Hoelter, 



1983, p. 31). 
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lQEC§!D§Dt§i EaCffl 1 Indices. 

2 2 2 2 

7) Incremental Form 1 index for X = BBI « (X - X )/ (X ) where 

o T o 

the subscripts o and T refer to the null and target models (see 

Bentler & Bonett, 1980). 

1 2 

8) Incremental Form 1 index for X /df (MBl) = 

2 2 2 

L (X /df ) - (X /df )] /[ (X /df 1 
^ o o T T 0 0 

9) Incremental Form 1 index -for RMS = (RMS - RMS )/(RMS ) 
J o T o 

10) Incremental Form 1 index for CN = (CN - CN )/(CN ) 

T o T 

Incrgmgntal Fgnn 2 Indices^ 

1 2 

11) Incremental Form 2 index for X (MB2) « 

2 2 2 
(X - X )/ (X - df ) where df is the 
o T o T T 

2 

expected value of X when the target model is true. 

2 

12) Incremental Form 2 index for X /df « TLI = 

2 2 2 

I (X /df ) - (X /df )3 /[ (X /df - 13 
0 0 T T o o 

2 

where 1 is the expected value of X /df when the target model is true. 



1 

We know of no previous descriptions of these goodness of fit indicators. 
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Table 1 

Mean and Standard grzqr iSEi q£ Ig Goodness g£ Fit Indicators For Seven Samele 
Sizes in EQUC BstS §§ts ^ 

Sample Size^:; 



n=25 n=50 n=100 n»200 nc400 n=800 nel600 Total F-ratio 
Indicator 



1) 


2 

X 


















1 


M 

SE 


35.55 
4.46 


34.06 
2.71 


37.65 
3.18 


34.52 
3.19 


48.43 
1.75 


49.26 
3.82 


70.60 
4.60 


44.30 14.2tt 

1,95 


2 


M 

SE 


56.04 
4.49 


78.20 
4.47 


119.64 
7.09 


210.77 
5.95 


414.43 
13.12 


738.89 1439.03 
25.76 29.04 


435.53 983. 2«« 
56.42 


3 


M 

SE 


24.84 
2.30 


25.33 
1.84 


24.27 
2.55 


23.34 
1.95 


24.79 
1.86 


23.64 
3.18 


23.38 
2.59 


24.23 .1 
.86 


4 


M 

SE 


30.90 
2.27 


'3.03 
3.22 


35.52 
2.37 


45.05 
2.60 


62.00 
5.12 


95.30 189.81 
5.19 9.11 


70.23 140. 5«« 
6.63 


2) 


X /di 
















1 


M 

SE 


1.48 
.19 


1.42 
.11 


1.57 
.13 


1.44 
.13 


2.02 
.07 


2.05 
.16 


2.94 
.19 


1.85 14.2«« 
.08 


2 


M 

SE 


2.34 
.19 


2.26 
.19 


4.90 

.30 


8.78 
.25 


17.27 
.55 


30.79 
1.07 


59.96 
1.21 


18.20 983. 2«« 
2.38 


3 


M 

SE 


1.04 
.10 


1.06 
.08 


1.01 
.11 


.9" 
.08 


1.03 
.08 


.99 
.13 


.98 
.11 


1.01 .1 
.04 


4 


M 

SE 


1.29 
.09 


1.37 
.13 


i.^e 

.10 


1.88 
.11 


2.58 
.21 


3.97 
.22 


7.91 
.38 


2.93 140. 5«« 
.28 


3) 


Goodness of 


Fit Index (GFI) 












1 


M 

SE 


.638 
.030 


.735 
.025 


.847 
.014 


.«?11 
.006 


.927 
.005 


.961 
.003 


.970 
.002 


.856 60.1«« 
.015 


2 


M 

SE 


.279 
.020 


.398 
.020 


.430 
.020 


.4 ',9 
.009 


.451 
.010 


.474 
.011 


.478 
.006 


.422 21.6«« 
.009 


3 


M 

SE 


.781 
.011 


.880 
.005 


.938 
.006 


.967 
.003 


.983 
.001 


.991 
.001 


.996 
.001 


.934 231. 0«« 
.009 


4 


M 

SE 


.750 
.016 


.849 
.013 


.913 
.006 


.940 
.004 


.956 
.004 


.964 

,002 


.964 
.002 


.906 91.5* 
.009 


4) 


Adj 


usted Goodness of Fit Index (AGFI) 








1 


M 

SE 


.322 
.0f>6 


.503 
.047 


.714 
.027 


.833 
.011 


.865 
.010 


.927 
.006 


.944 
.003 


.729 60. U« 
.028 


2 


M 

SE 


-.352 
.038 


-.127 
.037 


-.068 
.037 


-.034 
.018 


-.028 
.019 


.013 
.020 


.020 
.011 


-.082 21.6«« 
.018 


3 


M 
SE 


.589 
.020 


.775 
.009 


.884 
.011 


.939 
.006 


.968 
.002 


.984 
.002 


.992 
.001 


.876 234.4* 
.017 


4 


M 
SE 


.513 
.030 


.718 
.024 


.837 
.012 


.887 
.007 


.919 
.007 


.934 
.004 


.934 
.004 


.823 91.2* 
.018 
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Table 1 continued 

Sample Sizes 

n=25 ns50 n«100 n=200 n«400 n=800 n=1600 Total F-ratio 
Indicator 



S) Root Mean Square Residual (RMS) 



1 


M 


.437 


.317 


.235 


.176 


.164 


.121 


.108 


.223 


53.2«« 




SE 


.030 


.022 


.017 


.010 


-007 

. W/ 


.005 


.004 






2 


M 


.031 


.025 


.024 


.022 


.021 


,021 


.020 


.023 


5. U 




SE 


.004 


.002 


002 






.001 


.001 






3 


M 


.189 


. 137 


.096 


.068 


.049 


.036 


.024 


.086 


141.lt 




SE 


.009 


-004 


• WW 


.004 


.002 


.002 


.002 


• w/ 




4 


M 


.246 


• 168 


.122 


.099 


.086 


.078 


.079 


.125 


62.87 




SE 


.014 


013 


Ofi"5 




.005 


.003 


.002 






b) 


Critical N 


















1 


M 


31.2 


61.6 


109.6 


P^l.l 


324.9 


680.6 619.4 


337.8 


80. 9» 




SE 


3.7 


6.8 


8.6 


19.8 


11.6 


74.9 


59.9 


40.0 




2 


M 


18.6 


26.3 


34.2 


38.0 


38.8 


43.5 


44.3 


34.8 


42. 2n 




SE 


1.4 


1.8 


1.9 


1.1 


1.2 


1.6 


0.9 


1.2 


3 


M 


41.0 


79.9 


195.3 


357.7 


667.5 1456.5 3114.1 


844.6 


30. 2« 




SE 


3.1 


5.8 


41.8 


34.2 


62.7 


124.9 515.0 


144.1 




4 


M 


33.0 


65.5 


113.8 


178.0 


268.2 


335.7 334.8 


189.9 


78. 9« 




SE 


2.B 


8.5 


7.6 


10.5 


23.7 


13.9 


15.1 


14.9 





7) X Bentler-Bonett Index (K*I) 



1 


M 


.777 


.883 


.930 


.966 


.975 


.987 


.991 


.930 82. 1« 




SE 


.020 


.007 


.006 


.003 


.001 


.001 


.001 


.009 




2 


M 


.820 


.856 


.893 


.900 


.902 


.912 


.913 


.885 


20. St 




SE 


.016 


.009 


.005 


.003 


.003 


.003 


.002 


.005 




3 


M 


.633 


.745 


.830 


.907 


.948 


.974 


.987 


.861 


67.3»» 




SE 


.028 


.022 


.022 


.008 


.004 


.003 


.001 


.016 




4 


M 


.615 


.710 


.775 


.846 


.883 


.915 


.913 


.808 


81. 6« 




SE 
2 


.011 


.023 


.014 


.009 


.011 


.004 


.004 


.014 




8) 


X /df 


Incremental Fit Index 


1 (MBl) 












1 


M 


.666 


.824 


.894 


.949 


.'►62 


.981 


.986 


.895 


82. U» 




SE 


.030 


.011 


.009 


.005 


.002 


.002 


.001 


.014 




2 M 


.730 


.784 


.839 


.350 


,850 


.867 


.870 


.827 


20. 5« 




SE 


.024 


.014 


.008 


.004 


.005 


.0'05 


.003 


.007 




3 


M 


.451 


.617 


.745 


.861 


.921 


.961 


.981 


.791 


67.3«« 




SE 


.042 


.033 


.032 


.012 


.006 


.005 


.002 


.024 




4 H 


.423 


.564 


.662 


.769 


.824 


.873 


.869 


.712 


81.6* 




SE 


.016 


.035 


.021 


.013 


.016 


.007 


.006 


.020 




9) 


Incremental 


RMS 
















1 


M 


.731 


.808 


.862 


.894 


.897 


.924 


.932 


.864 


84. 9« 




SE 


.013 


.010 


.009 


.006 


.003 


.003 


.003 


.009 




2 


M 


.809 


.797 


.834 


.842 


.848 


.853 


.852 


.833 


3.U 




SE 


.020 


.023 


.010 


.006 


.004 


.003 


.003 


.005 




3 


M 


.517 


.607 


.681 


.754 


.823 


.868 


.912 


.738 49. 8«* 




SE 


.031 


.031 


.024 


.017 


.007 


.008 


.006 


.018 




4 


M 


.467 


.575 


.628 


.702 


.734 


.770 


.758 


.662 


35.5«» 




SE 


.026 


.031 . 


.020 


.011 


.013 


.006 


.007 


.014 
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Table 1 continued 



Sample Sizes 



n=25 
Indicator 


n=50 


n=100 


n=200 


n 3^400 

1 1 ~ w 




nsl AOO 
n— low 


1 Ouax 


r rati 


10) Incremental 


Critical N 














1 M 
SE 


.667 
.030 


.822 
.011 


.893 
.009 


.949 
.005 


.001 


• 002 


. #00 

.001 


.894 
.014 


84.6«« 


2 M 
SE 


.709 
.024 


.769 
.014 


.825 
.008 


.004 


QAi 
• OH I 

.005 


.005 


.003 


.814 23. 6» 
.008 


3 M 
SE 


.479 
.038 


.637 
.031 


.760 
.030 


.869 
.011 


.926 
.005 


.964 
.005 


.982 
.002 


.802 
.022 


70.6U 


4 M 

SE 


.451 
.015 


.588 
.033 


.681 
.020 


.781 
.013 


.839 
.012 


.880 
.006 


.877 
.006 


.728 
.019 


89.0»» 


11) Tucker-Lewis 


Index 


(TU) 














1 M 

SE 


.877 
.048 


.946 
.017 


.959 
.009 


.984 
.005 


,981 
.001 


*990 
.001 


.991 
.001 


.961 
.009 


4.2» 


2 M 
SE 


.825 
.026 


.839 
.014 


.867 
.008 


.864 
.004 


.860 
.005 


• Of * 

.005 


• o/ ^ 

.003 


.856 
.005 


2.2 


3 M 
SE 


1.013 
.106 


.975 
.041 


.988 
.035 


1.010 
.017 


.998 
.006 


1.001 
.005 


1.001 
.002 


.998 
.017 


. 1 


4 M 

SE 

2 

12) X 


.852 
.093 


.860 
.075 


.862 
.027 


.878 
.013 


.es4 

.017 


.902 
.007 


.884 
.006 


.874 
.017 


.1 


Incremental Fit 




(I1B2) 












1 M 

SE 


.924 
.029 


.966 
.011 


.974 
.006 


.989 
.003 


.001 


. TTO 

.001 


OOA 
« TT*t 

.001 


.975 
.005 


4.2* 


2 M 
SE 


.888 
.017 


.895 
.009 


.912 
.006 


.910 
.003 


.907 

. 003 


.914 

. V/V/O 


.914 


.906 
.003 


1.7 


3 M 
SE 


1.001 
.048 


.986 
.023 


.994 
.021 


1.010 
.011 


.999 
.004 


1.001 
.003 


1.000 
.001 


.998 
.008 


.1 


4 M 

SE 
2 a 


.910 
.042 


.915 
.040 


.916 
.016 


.923 
.008 


.924 
.011 


.935 
.004 


.923 
.004 


.921 
.009 


.1 



Null 

1 M 
SE 

2 M 
SE 

3 M 
SE 

4 M 

SE 



156.3 
8.3 

315.0 
11.5 

68.1 
4.0 

80.3 
5.5 



288.6 
15.5 

546.9 
17.4 

102.6 
6.9 

114.2 
7.2 



533.4 
20.3 



1024.2 
23.5 



1942.3 
48.2 



3820.7 
38.7 



7593. 
66. 



1111.4 


2102.4 


4213. 1 


8365.8 


16565. 1 


30.7 


31.4 


36.8 


53.3 


70.3 


147.3 


255.8 


472.6 


915.5 


1797.8 


5.0 


14.4 


12.9 


14.4 


27.9 


158.9 


296.7 


543.4 


1127,6 


2179.2 


8.2 


18.6 


12.7 


26.4 


33.6 



2194.8 5341.*? 
301.3 

4745.6 2134U* 
660.4 



537.1 
70.3 



1915»» 



642.9 1665. »« 
86.1 



9 2 

The X for the null model is presented to illustrate the substantial effect 
that sample size has on its value. 
* p < .05; « p < .01. 
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Figure 1. The sample size effect for selected fit indices in each of the 

four data sets. Since the form of the sample size effect was similar for X 

2 

and X /df , for BBI and MB2, and for TLX and MB2, only the first of each of 
these pairs is presented. However, the values for the fit indices not 
included in Figure 1 are presented in Table 1. 
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